This disclosure relates to an image processing method which uses a projector and a camera and in which a combination of an image projected by the projector and written information added to the projected image by an analogue device is stored as digital information. Such an image processing method can be used for software, which is implemented in personal computers (PC) in conjunction with a projector and a camera; projectors including a camera and a processor; mobile devices operating in conjunction with a projector; and mobile devices having a projector function.
Recently, projectors are broadly used, and meetings in which plural persons observe the same projected material (image) and discuss the material are often held. In such meetings, analogue images (i.e., additional information) are often written manually in the projected material. For example, there is a case in which points of discussion are manually written in the image projected on a screen (such as a whiteboard), or a sticky note bearing such an analog image thereon is attached to a screen. In this regard, there is a need such that such additional information is stored in a PC as digital information.
In attempting to fulfill such a need, JP-2004-239967-A discloses a projector capable of storing additional information (such as characters and figures) written on a screen such as a whiteboard.
When storing such additional information, a problem such that the image has perspective distortion tends to be caused. Specifically, when a projected image is photographed with a CCD (charge coupled device) camera, the photographing direction is not perpendicular to the surface of the projected image (i.e., the surface of the screen), and therefore perspective distortion is caused, i.e., the photographed image has a trapezoidal shape as illustrated in FIG. 12 although it is preferable for the photographed image to have a rectangle shape. Therefore, it is preferable to correct such perspective distortion of the photographed image.
Correction of perspective distortion itself is a popular technique. For example, as illustrated in FIG. 13, a method, in which a linear transformation matrix is calculated so that a trapezoidal shape is changed to a desired rectangle shape, and the matrix is used for correcting the tilt of the image, is popularly used for perspective distortion correction. The matrix is represented by the following equation, and has eight unknowns. Therefore, the solution can be obtained by using a simultaneous equation (1) having eight unknowns.
                              [                                                                      x                  t                  ′                                                                                                      y                  t                  ′                                                                                    1                                              ]                =                              [                                                                                m                    0                                                                                        m                    1                                                                                        m                    2                                                                                                                    m                    3                                                                                        m                    4                                                                                        m                    5                                                                                                                    m                    6                                                                                        m                    7                                                                    1                                                      ]                    ⁡                      [                                                                                x                    t                                                                                                                    y                    t                                                                                                1                                                      ]                                              (        1        )            
For example, in a case illustrated in FIG. 13, by specifying the four points (xn, yn) before correction and the four points (x′n, y′n) after correction, four equations can be obtained for each of x and y, i.e., eight equations can be obtained, thereby making it possible to obtain all the matrixes. In this regard, the suffix n (0≦n≦4) means the index number of a point.
By substituting the four points to the equation, the following simultaneous equation (2) can be obtained.
                                          [                                                                                x                    0                    ′                                                                                                                    x                    1                    ′                                                                                                                    x                    2                    ′                                                                                                                    x                    3                    ′                                                                                                                    y                    0                    ′                                                                                                                    y                    1                    ′                                                                                                                    y                    2                    ′                                                                                                                    y                    3                    ′                                                                        ]                    ⁡                      [                                                                                x                    0                                                                                        y                    0                                                                    1                                                  0                                                  0                                                  0                                                                                            -                                              x                        0                                                              ⁢                                          x                      0                      ′                                                                                                                                  -                                              x                        0                        ′                                                              ⁢                                          y                      0                                                                                                                                        x                    1                                                                                        y                    1                                                                    1                                                  0                                                  0                                                  0                                                                                            -                                              x                        1                                                              ⁢                                          x                      1                      ′                                                                                                                                  -                                              x                        1                        ′                                                              ⁢                                          y                      1                                                                                                                                        x                    2                                                                                        y                    2                                                                    1                                                  0                                                  0                                                  0                                                                                            -                                              x                        2                                                              ⁢                                          x                      2                      ′                                                                                                                                  -                                              x                        2                        ′                                                              ⁢                                          y                      2                                                                                                                                        x                    3                                                                                        y                    3                                                                    1                                                  0                                                  0                                                  0                                                                                            -                                              x                        3                                                              ⁢                                          x                      3                      ′                                                                                                                                  -                                              x                        3                        ′                                                              ⁢                                          y                      3                                                                                                                    0                                                  0                                                  0                                                                      x                    0                                                                                        y                    0                                                                    1                                                                                            -                                              x                        0                                                              ⁢                                          y                      0                      ′                                                                                                                                  -                                              y                        0                                                              ⁢                                          y                                              0                        ′                                                                                                                                          0                                                  0                                                  0                                                                      x                    1                                                                                        y                    1                                                                    1                                                                                            -                                              x                        1                                                              ⁢                                          y                      1                      ′                                                                                                                                  -                                              y                        1                                                              ⁢                                          y                                              1                        ′                                                                                                                                          0                                                  0                                                  0                                                                      x                    2                                                                                        y                    2                                                                    1                                                                                            -                                              x                        2                                                              ⁢                                          y                      2                      ′                                                                                                                                  -                                              y                        2                                                              ⁢                                          y                                              2                        ′                                                                                                                                          0                                                  0                                                  0                                                                      x                    3                                                                                        y                    3                                                                    1                                                                                            -                                              x                        3                                                              ⁢                                          y                      3                      ′                                                                                                                                  -                                              y                        3                                                              ⁢                                          y                                              3                        ′                                                                                                                  ]                          ⁡                  [                                                                      m                  0                                                                                                      m                  1                                                                                                      m                  2                                                                                                      m                  3                                                                                                      m                  4                                                                                                      m                  5                                                                                                      m                  6                                                                                                      m                  7                                                              ]                                    (        2        )            
Next, an inverse matrix of the thus obtained matrix (the left matrix of the right-hand side of formula (1)) is obtained, followed by multiplying each of the after-correction pixel positions x′t, y′t by the inverse matrix to determine the pixel values of the before-correction positions xt, yt. By using the thus obtained pixel values as after-correction pixel values, distortion correction is performed.
As mentioned above, in order to perform popular correction of trapezoidal shape distortion, coordinates of the four points before correction and coordinates of the four points after correction have to be specified. In this regard, JP-2004-239967-A describes perspective distortion correction, but does not describe a specific method for correcting perspective distortion.
JP-2001-084365-A and JP-2006-129511-A have disclosed specific methods for correcting perspective distortion in which the points are manually designated. Since it is burden on users to manually designate points, methods in which the points are automatically detected have been proposed. For example, JP-2008-014976-A discloses a method in which the subject to be photographed is limited to a paper sheet, and the four sides of the paper sheet are determined by edge detection to determine four points before correction from the intersections of the four sides. JP-2008-014976-A does not describe the method for specifying the four points after correction, but it is considered that the four points after correction can be easily determined because the image after correction has a rectangle shape, and the aspect ratio of the rectangle shape is generally constant.
This automatic perspective distortion correction is convenient for users, but the subject to be photographed has to be clarified. For example, when the subject to be photographed is a whiteboard and the whiteboard is photographed with a camera, the four sides of the whiteboard cannot be necessarily determined because the shape of the edges of the whiteboard is unknown. In addition, the aspect ratio of whiteboard is not necessarily constant. Therefore, it is hard to automatically input the aspect ratio of the rectangle shape after correction.